Live Music Archive
Librivox Free Audio
Metropolitan Museum
Cleveland Museum of Art
Internet Arcade
Console Living Room
Open Library
American Libraries
TV News
Understanding 9/11The Wayback Machine's First Crawl 1996
Video Item Preview
Share or Embed This Item
In October of 1996, engineers at the the San Francisco-based Internet Archive launched their first web crawlers, taking snapshots of web pages. At the time, the World Wide Web was only 2.5 terabytes in size. In 1996, it was still impossible to predict how large the World Wide Web might become.
Even in those early days of the Web, broken links (404 errors) were a growing problem, and it was clear that most Web pages were short-lived. Brewster Kahle and Bruce Gilliat invented a system for archiving Web pages before they vanished. The tools for this project were not terribly sophisticated; they were essentially PC applications built to capture entire websites by following the links from the main page.
In this video, shot by Marc Weber and Kevin Hughes for the Web History Project, Brewster Kahle explains his hardware and process, while the first crawl is underway.
25 years later, the Wayback Machine--which launched as a public search engine of web pages in 2001--has captured some 588 billion web pages by working with 800+ partners around the world.
In 2021, Internet Archive founder, Brewster Kahle, reflects back on the most surprising advancement of his early innovation, the Wayback Machine.
Produced by Wendy Hanamura
Edited by Kristin Tieche
Videographer: Brad Shirakawa
Archival footage courtesy of Marc Weber and Kevin Hughes, The Web History Project, 1996, all rights reserved.
- Addeddate
- 2021-08-06 23:41:23
- Ccnum
- asr
- Closed captioning
- yes
- Color
- color
- Identifier
- wayback-machine-1996
- Scanner
- Internet Archive HTML5 Uploader 1.6.4
- Sound
- sound
- Whisper_asr_module_version
- 20241218.01
- Year
- 2021
comment
Reviews
(33)
Reviewer:
chemenv
-
favoritefavoritefavoritefavoritefavorite -
October 22, 2021
Subject: WOW!! Well done guys!
Subject: WOW!! Well done guys!
Indeed what you achieved is extraordinary!
If we consider that the web is the Library (the 'library' of libraries or whatever one can call it - but a ... 'library' where some of its 'books' may be destroyed/lost/whatever, as it happens with actual libraries), building the Wayback Machine is the only solution of preserving it...! For ever, if one can tell!!
And I have been 'saved' in the past by your machine when a 'book' was lost!
Well done guys!!
And of course you were and are visionaries!
Well done and keep up the good work!
If we consider that the web is the Library (the 'library' of libraries or whatever one can call it - but a ... 'library' where some of its 'books' may be destroyed/lost/whatever, as it happens with actual libraries), building the Wayback Machine is the only solution of preserving it...! For ever, if one can tell!!
And I have been 'saved' in the past by your machine when a 'book' was lost!
Well done guys!!
And of course you were and are visionaries!
Well done and keep up the good work!
Reviewer:
DMan4DJob
-
favoritefavoritefavoritefavoritefavorite -
October 18, 2021 (edited)
Subject: THESE GUYS Are TRUE 'Visionaries'!
Subject: THESE GUYS Are TRUE 'Visionaries'!
Move over, Steve Jobs! WAY over! Further than that! OK. Great. Brewster Kahle and Bruce Gilliat are ACTUAL, LITERAL 'Visionaries'. THEY "changed the world
...
🗺". Not that hack that only specialized in manipulating the masses with slick talk and psychological slight-of-hand. Thank you, Brewster & Bruce! Your foresight to preserve something so precious and technologically poignant! 🤗🏆👌🏾
"YOU'RE SO COOL, BREWSTER!!" 🦇🧛♂️
"YOU'RE SO COOL, BREWSTER!!" 🦇🧛♂️
Reviewer:
jsmog310
-
favoritefavoritefavoritefavoritefavorite -
October 12, 2021
Subject: Very Important Work
Subject: Very Important Work
As a librarian, you are all heroes of mine. If not for your foresight, so much would have been lost. I can vouch that using the Wayback Machine, I found
...
a treasure trove of photographs and documents regarding the co-op I lived in at UC Berkeley in the early 1980s, which had been placed online and then neglected by a computer-savvy resident around 2000. Everyone thought they were gone forever, but thanks to your hard work, I found them and recovered them for our posterity. Of course, we uploaded them to Facebook and I left Facebook, so...
Reviewer:
LuisAlfredo92
-
favoritefavoritefavoritefavoritefavorite -
October 12, 2021
Subject: Amazing
Subject: Amazing
This video is completely amazing, it shows how the time has passed and they are still working on one of the greatest internet projects
Reviewer:
Riparian9857
-
favoritefavoritefavoritefavoritefavorite -
October 11, 2021
Subject: Hurrah for Brewster Kahle!
Subject: Hurrah for Brewster Kahle!
Imagine the labour intensity of this project when it was limited to only 20GB of storage... and data had to be transferred to TAPES! To have developed
...
the Internet Archive from such tiny beginnings into the immense resource we know today is a truly amazing achievement. Humanity thanks you, people of the Wayback Machine. <3
Reviewer:
Torda Gábor
-
favoritefavoritefavoritefavoritefavorite -
October 11, 2021
Subject: Time
Subject: Time
He had the vision. I was just starting to use internet in that time.
Reviewer:
DevonSeaMoor
-
favoritefavoritefavoritefavoritefavorite -
October 9, 2021
Subject: For better or for worse? Internet? Hold on... think again...
Subject: For better or for worse? Internet? Hold on... think again...
The Archive is a blessing to humanity, a great service to the freedom of people gaining knowledge.
I remember how a villager in remote Africa has found ... access to the internet of things, and managed to study, online, working his way up to a master's degree. That's how beneficial Internet is to all those who are interested to learn, know, and act on what they know. Hopefully in integrity.
For some years, I've been critical about data-collecting, storage of information about human beings, and their where-abouts and what-abouts. After all, China is an example of a fully covered social credit system, and every smartphone buyer must present its face-picture, so that all content of the phone can be verified as coming from that person. When someone is caught in going against what the government dictates, the bank account of that person can be closed with a push on a button. With social ostracising in tow. In short, there's huge control. No privacy.
And as some of you know the Edward Snowdon Treaty, freedom can never be granted by others, as part of some form of trade. Freedom is an innate human right and although we've got a way to go, that "grey space in the middle" defined as the essence of the 21st century by David Bowie, that's what the Global Privacy Treaty is about, as I perceive it
Nowadays, still a critical thinker, for which I'm grateful, I believe that even when I know that knowledge can be used for better or for worse, that certain individuals, or organisations can dig in the archives here, finding info of people, and use it to that person's disadvantage, that the actual presence of this archive offers us a chance to rectify this wrong doing. On the long run, for sure.
Just like in the case of the rape crime where a couple of youngsters was convicted, and later liberated when the real perpetrator showed up. For over a decade, I'm a member here, and I've found valuable content, of all sorts. Still, many Dutch people don't know that Archives exist.
This Dutch review writer brings change in that condition 😉
I remember how a villager in remote Africa has found ... access to the internet of things, and managed to study, online, working his way up to a master's degree. That's how beneficial Internet is to all those who are interested to learn, know, and act on what they know. Hopefully in integrity.
For some years, I've been critical about data-collecting, storage of information about human beings, and their where-abouts and what-abouts. After all, China is an example of a fully covered social credit system, and every smartphone buyer must present its face-picture, so that all content of the phone can be verified as coming from that person. When someone is caught in going against what the government dictates, the bank account of that person can be closed with a push on a button. With social ostracising in tow. In short, there's huge control. No privacy.
And as some of you know the Edward Snowdon Treaty, freedom can never be granted by others, as part of some form of trade. Freedom is an innate human right and although we've got a way to go, that "grey space in the middle" defined as the essence of the 21st century by David Bowie, that's what the Global Privacy Treaty is about, as I perceive it
Nowadays, still a critical thinker, for which I'm grateful, I believe that even when I know that knowledge can be used for better or for worse, that certain individuals, or organisations can dig in the archives here, finding info of people, and use it to that person's disadvantage, that the actual presence of this archive offers us a chance to rectify this wrong doing. On the long run, for sure.
Just like in the case of the rape crime where a couple of youngsters was convicted, and later liberated when the real perpetrator showed up. For over a decade, I'm a member here, and I've found valuable content, of all sorts. Still, many Dutch people don't know that Archives exist.
This Dutch review writer brings change in that condition 😉
Reviewer:
BenZen528
-
favoritefavoritefavoritefavoritefavorite -
October 9, 2021
Subject: Loss of history - or the wilful redaction of it
Subject: Loss of history - or the wilful redaction of it
I have been helped by The Internet Archive on several occasions: When a large website in my organisation was hacked and a lot of the content disappeared
...
– or when websites were closed or changed due to various circumstances. In these cases, I have been able to recover valuable content.
Another case was this: Some years ago, I watched the disturbing documentary “The Central Park Five” (2012) about the unjust conviction of five male teenagers after a brutal rape of a female jogger in Central Park in 1989. They were sentenced to six to twelve years in prison. (This was after Donald J. Trump ran a full-page ad in Daily News urging: Bring back the death penalty. In 2002 their convictions were overturned after a convicted serial rapist confessed to this rape also and his DNA matched him to the crime.)
I wanted to learn more about the people involved in the case, so I did a Google search for the Assistant District Attorney Elizabeth Lederer and found that she was teaching at Columbia University.
I made a search on The Internet Archive and found that the bio of Elizabeth Lederer on January 15, 2013 read:
“Ms. Lederer is an Assistant District Attorney, New York County. She served as a line assistant trying misdemeanor and felony cases, handling motion practice and grand jury presentations, specializing in sex crimes and homicide cases (1979-85); Worked as Senior Trial Counsel trying homicide cases and high profile cases including the “Central Park Jogger” case and …”
Then, on 24 April 2013 Elizabeth Lederer’s telephone number disappeared and her bio on Columbia University’s website was redacted/reduced to a version that omitted the reference to ‘the “Central Park Jogger” case’.
Why this redaction? Well, something happened between these two dates:
On Wikipedia - https://en.wikipedia.org/wiki/Central_Park_jogger_case#Representation_in_other_media – you can read that “On September 12, 2012, attorneys for New York City subpoenaed the production company [of the documentary “The Central Park Five” mentioned above] for access to the original footage in connection with its defense of the 2003 federal civil lawsuit brought against the city by three of the convicted youths. Celeste Koeleveld, the city's executive assistant corporation counsel for public safety, justified the subpoena on the grounds that the film had "crossed the line from journalism to advocacy" for the wrongfully convicted men. In February 2013, U.S. Judge Ronald L. Ellis quashed the city's subpoena.”
This is one example of the benefits The Internet Archive can deliver to the World: help uncovering the truth.
After a new miniseries about the case “When They See Us”, Ms. Lederer resigned from Columbia Law School in 2019.
Another case was this: Some years ago, I watched the disturbing documentary “The Central Park Five” (2012) about the unjust conviction of five male teenagers after a brutal rape of a female jogger in Central Park in 1989. They were sentenced to six to twelve years in prison. (This was after Donald J. Trump ran a full-page ad in Daily News urging: Bring back the death penalty. In 2002 their convictions were overturned after a convicted serial rapist confessed to this rape also and his DNA matched him to the crime.)
I wanted to learn more about the people involved in the case, so I did a Google search for the Assistant District Attorney Elizabeth Lederer and found that she was teaching at Columbia University.
I made a search on The Internet Archive and found that the bio of Elizabeth Lederer on January 15, 2013 read:
“Ms. Lederer is an Assistant District Attorney, New York County. She served as a line assistant trying misdemeanor and felony cases, handling motion practice and grand jury presentations, specializing in sex crimes and homicide cases (1979-85); Worked as Senior Trial Counsel trying homicide cases and high profile cases including the “Central Park Jogger” case and …”
Then, on 24 April 2013 Elizabeth Lederer’s telephone number disappeared and her bio on Columbia University’s website was redacted/reduced to a version that omitted the reference to ‘the “Central Park Jogger” case’.
Why this redaction? Well, something happened between these two dates:
On Wikipedia - https://en.wikipedia.org/wiki/Central_Park_jogger_case#Representation_in_other_media – you can read that “On September 12, 2012, attorneys for New York City subpoenaed the production company [of the documentary “The Central Park Five” mentioned above] for access to the original footage in connection with its defense of the 2003 federal civil lawsuit brought against the city by three of the convicted youths. Celeste Koeleveld, the city's executive assistant corporation counsel for public safety, justified the subpoena on the grounds that the film had "crossed the line from journalism to advocacy" for the wrongfully convicted men. In February 2013, U.S. Judge Ronald L. Ellis quashed the city's subpoena.”
This is one example of the benefits The Internet Archive can deliver to the World: help uncovering the truth.
After a new miniseries about the case “When They See Us”, Ms. Lederer resigned from Columbia Law School in 2019.
Reviewer:
Phycom
-
favoritefavoritefavoritefavoritefavorite -
October 8, 2021
Subject: The Wayback Machine is SO useful
Subject: The Wayback Machine is SO useful
Thank you archive.org for the Wayback Machine. Time and time again I have had to access old webpages from books (IT books especially) that reference websites
...
that are 5 years old and thanks to you I can still see the contents! I will always be a fan of the Wayback Machine!
Reviewer:
slavomira
-
favoritefavoritefavoritefavoritefavorite -
October 8, 2021
Subject: Much obliged!!
Subject: Much obliged!!
The memory is a complex term. And there is always “lack of memories”.
I think that I always have to keep this in mind.
Great job! 😊 I should say ... that in 2021, English is Esperanto language for today.
😊 I check it, from time to time, with Cyrillic languages. In hope, that I do a little feedback for the crawling.
With sincere best feelings from Bulgaria!
I think that I always have to keep this in mind.
Great job! 😊 I should say ... that in 2021, English is Esperanto language for today.
😊 I check it, from time to time, with Cyrillic languages. In hope, that I do a little feedback for the crawling.
With sincere best feelings from Bulgaria!
Reviewer:
WinnieSchneider
-
favoritefavoritefavoritefavoritefavorite -
October 7, 2021
Subject: The exciting journey continues …
Subject: The exciting journey continues …
👏👏👏 from good old Germany ,-)
Reviewer:
The-Martian
-
favoritefavoritefavoritefavoritefavorite -
October 7, 2021
Subject: Your website is about to change HISTORY
Subject: Your website is about to change HISTORY
Records from the past, uplaoded here, exposes the enemy of our souls that seeks our destruction. 150 years of censorship omes to a screeching halt here.
...
Praise the Lord!
Reviewer:
Bradford McCormick
-
favoritefavoritefavoritefavoritefavorite -
October 7, 2021
Subject: Thanks
Subject: Thanks
I was actively using the Wayback Machine at latest by 01 January 2003. I datestamp all my personal webpages and I wrote a blurb for Wayback Machine ona
...
page you saved: http://www.users.cloud9.net/~bradmcc/elsewheres.html with date on page: 01 January 2003. Anybody remember "Third Voice"? Keep up the good work, folks! That ISP decided web hosting was not profitable for them in 2020, so my website o 23 years no longer exists. I have a new one, which I will be archiving, at https://www.bmccedd.org. Je me souviens.
Reviewer:
Xatriya
-
favoritefavoritefavoritefavoritefavorite -
October 7, 2021
Subject: Great work
Subject: Great work
Really great work.... Hope this goes on infinitely across generations.
Reviewer:
Walrus Osaurus
-
favoritefavoritefavoritefavoritefavorite -
October 7, 2021
Subject: INCREDIBLE
Subject: INCREDIBLE
So amazing to see where this wonder started, thank you all.
Reviewer:
tackyou
-
favoritefavoritefavoritefavoritefavorite -
October 7, 2021
Subject: Legend
Subject: Legend
Thank you for all your work, without you a lot would be lost to time.
Reviewer:
Blake00
-
favoritefavoritefavoritefavoritefavorite -
October 7, 2021
Subject: thank you
Subject: thank you
This is great. I love this place so much. Been find long lost websites here for many years. A few years back I started contributing and preserving rare
...
video game magazines here too. Now I've started preserving compressed copies of rare/lost classic Civilization series scenarios and mods in there as I think this place has great potential for creating fan mod libraries similar to ModDB.
Reviewer:
Yesan Sellan
-
favoritefavoritefavoritefavoritefavorite -
October 7, 2021
Subject: Small Beginnings
Subject: Small Beginnings
Great Things started in a small way. The wayback machine is a great idea. Congratulations. IA has created huge repository of webpages.
Reviewer:
brewster
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: Wayback Machine humble beginnings
Subject: Wayback Machine humble beginnings
Strange to watch yourself after 25 years-- and that I called it the Wayback Machine even then... (that idea was from Z Smith, who ran the technology
...
back then).
fun.
fun.
Reviewer:
MavenConrad
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: Remarkable
Subject: Remarkable
Having the ability to view what has been is an extraordinary accomplishment! Thank you Internet Archive for all the brilliant work you have brought to
...
the world.
Reviewer:
G.L.Tyrebyter
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: The Alexandria library of ephemera.
Subject: The Alexandria library of ephemera.
The internet is a huge technological leap for access to information. Unfortunately it is all ephemeral. Sites come and go and links and references vanish.
...
The wayback machine has kept these resources alive and available. I have been a long time donor to the Internet Archives. I’d hate to see it all lost.
Reviewer:
Point_Barrow
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: Nice
Subject: Nice
Visionary. The humble beginnings reminded me of how thrilled I was with my Sinclair ZX-81 16KB RAM pack or when I bought my first 10MB hard drive. Keep
...
on storing!
Reviewer:
Charvella
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: Historic glimpse of brilliance.
Subject: Historic glimpse of brilliance.
Recording the history of recording the history of the www.
Reviewer:
Maazin Althaf
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: Amazing!
Subject: Amazing!
This guy was 25 years ahead of his time!
Reviewer:
Foulowe59_
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: Incredible
Subject: Incredible
Those were some really fascinating's old takes(like always)!
Reviewer:
Flippers
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: One of the older web sites recorded here...
Subject: One of the older web sites recorded here...
I started my web site (my business's site, so not including its' name here) in 1996 and the earliest record of it on the wayback machine is April 26, 1997
...
- which is pretty good! So I think you may not have captured the entire web in October 1996.
I do greatly appreciate the effort and data access, and use the Wayback Machine regularly to track things down. How the heck do you keep at that data and have it available so quickly?
I do greatly appreciate the effort and data access, and use the Wayback Machine regularly to track things down. How the heck do you keep at that data and have it available so quickly?
Reviewer:
DarseZ
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: Fascinating
Subject: Fascinating
It must have not seemed like a very important thing was being done at the time, but as time passes, its' importance has only grown.
One wonders if the ... basic architecture in this early system still applies to the current system...
One wonders if the ... basic architecture in this early system still applies to the current system...
Reviewer:
Ka non
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: Amazing!!
Subject: Amazing!!
I owned a very stupid webpage in 1995, perhaps the year later was saved in this tapes....
Reviewer:
dosplatanus
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: Excellent
Subject: Excellent
Really good to know how the IA started.
Reviewer:
BCDC1949
-
favoritefavoritefavoritefavorite -
October 6, 2021
Subject: Thank You!
Subject: Thank You!
As an avid genealogist, I want to thank you for this proactive effort that realized the importance of preserving so much information from the onset of
...
the digital age. It's priceless and will become even more so as time goes by!
Reviewer:
Soppy Inc. official
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: Really interesting!
Subject: Really interesting!
Really interesting video!
Reviewer:
kaijian.ye
-
favoritefavoritefavoritefavoritefavorite -
October 6, 2021
Subject: This is a great journey
Subject: This is a great journey
This is a great journey
There are 33 reviews for this item. .
150,875 Views
196 Favorites
DOWNLOAD OPTIONS
IN COLLECTIONS
Internet Archive 25th Anniversary
Internet Archive Presents 
The Internet Archive 
Transcription Requested 
Uploaded by Wendy Hanamura on
